Determining points of parabolic curvature on surfaces of specular objects

ABSTRACT

Embodiments of the invention disclose a system and a method for determining points of parabolic curvature on a surface of a specular object from a set of images of the object is acquired by a camera under a relative motion between a camera-object pair and the environment. The method determines directions of image gradients at each pixel of each image in the set of images, wherein pixels from different images corresponding to an identical point on the surface of the object form corresponding pixels. The corresponding pixels having substantially constant the direction of the image gradients are selected as pixels representing points of the parabolic curvature.

FIELD OF THE INVENTION

This invention relates generally to determining points of paraboliccurvature, and more particularly to determining points of paraboliccurvature on surfaces of specular objects.

BACKGROUND OF THE INVENTION

Image invariants are properties of images of an object that remainunchanged with changes in parameters of a camera and/or illumination.For example, geometric invariants are related to apparent size ofdifferent parts of objects and are therefore equally valid for theobjects with any reflectance characteristics of the surface, includingdiffuse, specular and transparent objects. However, in order to use thegeometric invariants from the images of the object, pointcorrespondences across the images should be identified. Identifying thepoint correspondences from images of the diffuse object is a meaningfultask since the diffuse object has photometric features. But specularobject, i.e., the object having surface with mirror-like reflectance,does not have an appearance of its own, but rather present a distortedview of an environment surrounding the object.

Therefore, identifying the point correspondences using an image featuredescriptor of the specular object is challenging. The image featuredescriptor finds correspondences between reflections of the environment,which do not correspond to the same points on the surface of the object.Thus, there is a need to find photometric properties of the specularobject that are invariant to the surrounding environment.

Points of parabolic curvature are fundamental to perception of a shapeof the diffuse and/or the specular objects. Because these pointscorrespond to a geometric property of the surface, these points can thenbe used for a variety of machine vision tasks such as objectrecognition, pose estimation and shape regularization.

Accordingly, it is desired to determine photometric properties of theimages of mirror surfaces around points that exhibit parabolic curvaturewithout knowledge about shape of the surface of the specular objectand/or the surrounding environment.

SUMMARY OF THE INVENTION

It is an object of the subject invention to provide a method fordetermining a point of parabolic curvature on a surface of a specularobject.

It is further object of the invention to provide such a method thatdetermines the point of parabolic curvature without previous knowledgeabout shape of the object and/or surrounding environment.

It is further object of the invention to provide such a method thatdetermines the point of parabolic curvature based on uncalibrated imagesof the specular object.

It is further object of the invention to demonstrate that for thespecular object under a certain imaging setup, image derivatives at thepoints of parabolic curvature exhibit degeneracies independent of thesurrounding environment, and to provide a method that uses thedegeneracies to determine the point of parabolic curvature.

It is further object of the invention to provide a method for using thepoints of parabolic curvature for object detection, object recognitionand pose estimation.

The subject invention resulted from the realization that under relativemotion between a camera-object pair and the environment, features of theenvironment associated with each point of the surface of the objectchanges arbitrarily. However, points of parabolic curvature are tied tothe surface of the specular object, and hence, the direction of an imagegradient associated with these points of parabolic curvature aresubstantially constant.

One embodiment of the invention discloses a method for determiningpoints of parabolic curvature on a surface of a specular object from aset of images of the object is acquired by a camera under a relativemotion between a camera-object pair and the environment. The methoddetermines directions of image gradients at each pixel of each image inthe set of images, wherein pixels from different images corresponding toan identical point on the surface of the object form correspondingpixels. The corresponding pixels having substantially constant thedirection of the image gradients are selected as pixels representingpoints of the parabolic curvature.

Another embodiment discloses a system for determining points ofparabolic curvature on a surface of an object arranged in anenvironment, wherein the object is a specular object. The systemcomprises a camera-object pair configured to acquire a set of images ofthe object under a relative motion between a camera-object pair and theenvironment; and a processor configured to determine directions of imagegradients at each pixel of each image in the set of images, whereinpixels from different images corresponding to an identical point on thesurface of the object forms corresponding pixels, wherein the processorselects the corresponding pixels having substantially constant thedirections of the image gradients as pixels representing a point of theparabolic curvature.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a method for determining a point of aparabolic curvature of a surface of an object according to an embodimentof an invention;

FIG. 2 is a schematic of an image formation model according to oneembodiment of the invention;

FIGS. 3A-3C are examples of points of parabolic curvatures;

FIG. 4 is an example of a database of parabolic curvatures;

FIGS. 5A-5B is a visualization of image gradients; and

FIG. 6 is an example of environment reflections at a parabolic and a nonparabolic point.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 shows a method 100 for determining points 110 of paraboliccurvature on the surface of an object 115 arranged in an environment130. The object is a specular object having a parabolic curvature of thesurface. The points of parabolic curvature are the points of a surface,wherein the surface has no bending (curvature) along only one direction.For example, if the object is a cylinder, the surface of the cylinder isformed by the points at a fixed distance from an axis of the cylinder.The surface of the cylinder curves in the direction, e.g., perpendicularto the axis. However, the surface has no bending in the directionparallel to the axis along a height of the cylinder. Example of thepoints 310 of parabolic curvature of the surface 320 is shown in FIG.3A.

The points of parabolic curvature are determined based on a set 140 ofimages {I_(j)} of the object acquired by a camera 120 under a relativemotion 170 between a camera-object pair 125, and the environment. In thecamera-object pair, a relative pose between the camera and the object ispreserved. In different embodiments, the relative motion between thecamera-object pair and the environment is achieved by moving arbitrarilythe camera-object pair and/or the environment. A. For example, in oneembodiment, the environment is moved by projecting various patterns onthe environment using projectors. Steps of the method 100 are performedby a processor 101 as known in the art.

FIGS. 3B-C show an example of the set 140 of images and the points 110of the parabolic curvatures. Typically, the points of paraboliccurvature form a curve 330 on the surface of the object. The curve isreferred herein as the curve of parabolic curvature.

An image gradient describes a gradual blend of color from low to highintensity values of pixels in the images, such as in the images shown inFIGS. 5A-B. Mathematically, the image gradient is a two-variablefunction, e.g., the image intensity function. At each point of the imagethe function is determined by a gradient vector with components given byderivatives of the intensity values of pixels in horizontal and verticaldirections. At each point, the gradient vector points in the directionof largest possible intensity increase or decrease as indicated byarrows 510. Length of the gradient vector corresponds to rate of changein the value of the intensities in that direction.

Embodiments of the invention are based on a realization that directionsof the image gradients in the images of the specular objects areindependent of the environment at the points of parabolic curvature. Theimage gradients are always oriented in the same direction on the pointsof parabolic curvature. In particular, as shown in FIGS. 3B-3C, thedirections 340 of the image gradients are perpendicular 340 to the curveof the parabolic curvature.

The directions of the image gradients are determined 150 for all pixelsof images in the set of images such that the direction of the imagegradient at corresponding pixels 145 in the images forms a set ofdirections 155. The corresponding pixels are formed by pixels fromdifferent images corresponding to an identical point on the surface ofthe object, i.e., pixels having the same position in each image.

The corresponding pixels having substantially constant directions in theset of directions of the image gradients are selected as pixelsrepresenting a point of parabolic curvature. Accordingly, differentcorresponding pixels having substantially constant directions representdifferent points of parabolic curvature.

FIG. 6 shows fragments of images 140 of the specular object 115 havingthe points of parabolic and non-parabolic curvatures. The correspondingpixels 635 of the images 140 representing the point on non paraboliccurvature have arbitrarily directions 645 of the image gradients.However, corresponding pixels 625 of the images 140 representing thepoint of the parabolic curvature have constant directions 655 of theimage gradients.

Point of a Parabolic Curvature

In one embodiment, a shape of the specular object is defined using aMonge-Ampère equation(x,y,ƒ(x,y))=(x,ƒ(x))in a camera coordinate system where a function ƒ is twice continuouslydifferentiable. At each point on the surface, a curvature along a curveis defined as a reciprocal of a radius of an osculating circle. Theosculating circle is a circle whose center lies on the inner normalline, and curvature of the circle is the same as the curvature of thecurve. The principal curvatures are defined as minimum and maximumvalues of the curvature measured along various directions at each point.The product of the principal curvatures is defined as the Gaussiancurvature according to

$\begin{matrix}{\frac{{f_{xx}f_{yy}} - f_{xy}^{2}}{1 + f_{x}^{2} + f_{y}^{2}}.} & (1)\end{matrix}$

Points at which one of the principal curvatures is zero are the pointsof the parabolic curvature. If we defining a Hessian matrix at a point xof the surface as

$\begin{matrix}{{{H(x)} = {\frac{1}{2}\begin{bmatrix}f_{xx} & f_{xy} \\f_{xy} & f_{yy}\end{bmatrix}}_{(x)}},} & (2)\end{matrix}$then the points of the parabolic curvature have rankrank[H(x)]=1.

Image Formation for Specular Objects

FIG. 2 shows an image formation model according to one embodiment of theinvention. The images observed on the surface 210 of the specular objectwarp the surrounding environment 130. Some embodiments of the inventionmodel the camera 120 as an orthographic camera, i.e., all rays 220entering the camera are parallel to a principal direction.

The image gradient at a location 230 of the pixel x is∇ƒ=(ƒ_(x),ƒ_(y))^(T), and a normal 240 to the surface is

$\begin{matrix}{{n(x)} = {\frac{1}{\sqrt{1 + {{\nabla f}}^{2}}}{\begin{pmatrix}{- {\nabla f}} \\1\end{pmatrix}.}}} & (3)\end{matrix}$

A viewing direction v of the camera at each pixel is the same, i.e.,v=(0,0,1)^(T), where T is a transpose operator.

Under perfect mirror reflectance, the direction s(x) of the rayreflected onto the camera at a location of the pixel x is

$\begin{matrix}\begin{matrix}{s = {{2\left( {n^{T}v} \right)n} - v}} \\{= {\frac{1}{1 + {{\nabla f}}^{2}}{\begin{pmatrix}{{- 2}{\nabla f}} \\{1 - {{\nabla f}}^{2}}\end{pmatrix}.}}}\end{matrix} & (4)\end{matrix}$

The ray s(x) is a unit vector in three dimensional space, and,therefore, can be represented with spherical coordinates Θ(x)=(θ,φ),such that

S=[sin θ cos φ, sin θ sin φ, cos θ]^(T). The spherical coordinates canbe found via

$\begin{matrix}{{\tan\;{\phi(x)}} = \frac{f_{y}}{f_{x}}} & (5) \\{{\tan\;{\theta(x)}} = {\frac{2{{\nabla f}}}{1 - {{\nabla f}}^{2}}.}} & (6)\end{matrix}$

The environment observed by the camera at the location of the pixel x isdefined by an intersection of the environment and the ray in thedirection s(x) from the location of the pixel x, i.e., (x,ƒ(x))^(T).

In one embodiment, the environment is set at infinity. Accordingly, thedependence on the location of the pixels is suppressed, and theenvironment observed by the camera depends only on the gradient ∇ƒ ofthe image. The embodiment defines an environment map E:S²

R over a sphere S², and selects the Euler angle parameterization for thesphere. Under no inter-reflectance within the object, a forward imagingequation for the intensity I(x) observed at the location of the pixel xisI(x)=E(Θ(x))  (7)where Θ(x) is the Euler angle of the observed ray according to Equations(5) and (6).

Differentiating Equation (7) with respect to the pixel x, the imagegradients are determined according to

$\begin{matrix}{{{\nabla_{x}I} = {2\;{{H(x)}\left\lbrack \frac{\partial\Theta}{\partial{\nabla f}} \right\rbrack}^{T}{\nabla_{\Theta}E}}},} & (8)\end{matrix}$wherein ∇_(x)I are the image gradient at the location of the pixel x,and ∇_(Θ)E is a gradient of the environment with respect to sphericalcoordinates.

For the points of the parabolic curvature, the matrix H(x) is singular.Therefore, the observed image gradient in the direction of the zeroeigenvector is zero. Thus, the observed image gradient is oriented inthe direction of the eigen-vector corresponding to non-zero eigen valueof the matrix H(x). The matrix H(x) is a property of the surface and isindependent of the environment. So, the direction of the image gradientis independent of the environment and is substantially constant atpoints of parabolic curvature. Furthermore, the non-zero eigenvector isoriented in the direction perpendicular to the curve of paraboliccurvature. Therefore, at points of parabolic curvature, the direction ofimage gradients is independent of the environment and oriented in adirection perpendicular to the curve of parabolic curvature.

Invariant

As shown in FIG. 2, the surface (x,ƒ(x)) of the specular object issmooth, placed with the environment at infinity and observed by theorthographic camera. The image gradients at the points of the paraboliccurvature degenerate and take values along a single direction defined bythe shape of the surface. The degeneration is independent of theenvironment, and defined as an invariant according to the embodiments.

The invariant arises due to the principal direction of zero curvature atthe points of the parabolic curvature. By definition, an infinitesimalmovement on the surface along the principle direction does not changethe surface normal since the curvature is zero in that direction.Accordingly, the environment observed at the point of the paraboliccurvature depends only on the surface normal. Hence, an infinitesimaldisplacement on the image plane along the projection of the principledirection does not change the environment observed. Accordingly, theimage gradient along the principle is zero.

Mathematically, the invariant is expressed in various forms depending onthe embodiment. Based on Equation (8), parabolic curvature at the pointx₀ satisfies∇_(x) I(x ₀)=∥∇_(x) I(x ₀)∥v,  (9)where v is the eigenvector of the matrix H(x₀) with non-zero eigenvalue.Additionally or alternatively, some embodiments determine the points ofthe parabolic curvature using gradient autocorrelation matrix M(x)defined according to

$\begin{matrix}{{{M(x)} = {\sum\limits_{E}\;\left( {\left( {\nabla_{x}{I\left( {x;E} \right)}} \right)\left( {\nabla_{x}{I\left( {x;E} \right)}} \right)^{T}} \right)}},} & (10)\end{matrix}$wherein I(x; E) is the intensity observed at pixel x under theenvironment E(Θ). The summation of Equation (10) is determined over allpossible mapping points of the environment, however at points ofparabolic curvature,rank[M(x ₀)]=1.  (11)

In contrast with the parabolic surfaces, the gradient autocorrelationmatrix M(x) for elliptic and hyperbolic surfaces is full rank. For flatsurfaces, matrix H(x) is the zero matrix, the image gradients are zeroand the gradient autocorrelation has zero rank.

In one embodiment, the invariant does not take inter-reflections intoaccount. Inter-reflections alter the imaging process locally, andviolate the image formation mode. Similarly, resolution of the imagesaffects the embodiments of the invention. For images with lowresolution, the curvature of the surface observed in a single pixel candeviate from the parabolic curvature.

Selecting Points of Parabolic Curvature

As described above, embodiments of the invention select as the point ofparabolic curvature the point on the surface of the object correspondingto the pixel having a substantially constant direction of the imagegradient.

For example, one embodiment determines the gradient autocorrelationmatrix using image gradients determined at each image j in the set ofimages {I_(j)} according to

$\begin{matrix}{{{M(x)} = {\sum\limits_{j}{{\nabla_{x}{I_{j}(x)}}{\nabla_{x}{I_{j}(x)}^{T}}}}},} & (12)\end{matrix}$wherein j is the image in the set of images {I_(j)}, ∇_(x)I is the imagegradient, I(x) is an intensity observed at a location of a pixel x, T isthe transpose operator. Ratio of the eigenvalues of the gradientautocorrelation matrix is used as statistic to select the pixel x as thepoint of the parabolic curvature.

Pose of the Object

In one embodiment of the invention, the points of the paraboliccurvature facilitate determination of a pose of the object. The posedetermination method according the embodiment recovers three dimensional(3D) rotation and 3D translation parameters with respect to a predefinednominal pose of the object. If the camera is orthographic, then the poseof the object pose is recovered up to a depth ambiguity.

As shown in FIG. 4, templates 410 of parabolic curvatures are determinedby rotating the points of the parabolic curvature. In one embodiment, aparametric form and/or a 3D model of the object is acquired in advance.The 3D positions of the points of the parabolic curvature at objectcoordinates are recovered either analytically using a parametric form,or numerically using the 3D model of the object. During initialization,the templates are determined by rotating the points of paraboliccurvature with respect to a set of sampled 3D rotations and projectingvisible points to an image plane.

One variation of this embodiment considers storing only out-of-planerotations (θ_(x) and θ_(y)) in the database, because the rotation of theobject along principal axes (θ_(z)) of the camera results in an in-planerotation of the parabolic curvature points on the image plane. Therotations is performed by uniform sampling of the angles on a 2-sphere.

The pose of the object is determined by searching the templates andoptimal Euclidean transformation parameters s=(θ_(z), t_(x), t_(y)),which aligns the templates with the points of the parabolic curvature ofthe object. One embodiment uses a chamfer matching technique to measurea similarity between the templates and the points of the paraboliccurvatures. The template with the most similar parabolic curves to thepoints of the parabolic curvature is selected as the pose of the object.

Precision of the pose is limited by the discrete set of out-of-planerotations included into the database. In one embodiment, the pose isrefined using a combination of iterative closest point and Gauss-Newtonoptimization.

Recognition

The points of the parabolic curvature allow recognizing differentobjects in variable poses. The object recognition method is an extensionof the aforementioned pose determination method. For each object, thepose determination is repeated to recover best pose parameters. A classof the object is determined based on a minimum of the chamfer costfunction over different classes of the object.

Although the invention has been described by way of examples ofpreferred embodiments, it is to be understood that various otheradaptations and modifications may be made within the spirit and scope ofthe invention. Therefore, it is the object of the appended claims tocover all such variations and modifications as come within the truespirit and scope of the invention.

We claim:
 1. A method for determining points of parabolic curvature on asurface of an object arranged in an environment, wherein the object isspecular, comprising the steps of: obtaining a set of images of theobject acquired by a camera under a relative motion between acamera-object pair and the environment; determining directions of imagegradients at each pixel of each image in the set of images, whereinpixels from different images corresponding to an identical point on thesurface of the object form corresponding pixels; and selecting thecorresponding pixels having substantially constant the directions of theimage gradients as pixels representing points of the paraboliccurvature, wherein the steps of the method are performed by a processor.2. The method of claim 1, wherein the surface at the points of theparabolic curvature curves only in one direction.
 3. The method of claim1, wherein a relative pose between the camera and the object ispreserved during the relative motion.
 4. The method of claim 1, whereinthe relative motion between the camera-object pair and the environmentis achieved by moving arbitrarily at least one of the camera-object pairand the environment.
 5. The method of claim 1, wherein differentcorresponding pixels having the substantially constant directions of theimage gradients represent different points of the parabolic curvature.6. The method of claim 1, wherein the surface is twice continuouslydifferentiable.
 7. The method of claim 1, wherein the camera is anorthographic camera.
 8. The method of claim 1, wherein the environmentis set at infinity, such that the environment observed by the cameradepends only on the image gradients.
 9. The method of claim 1, furthercomprising: determining a gradient autocorrelation matrix M(x) accordingto${{M(x)} = {\sum\limits_{j}{{\nabla_{x}{I_{j}(x)}}{\nabla_{x}{I_{j}(x)}^{T}}}}},$wherein j indexes of the images in the set of images {I_(j)}, ∇_(x)I isthe image gradient, I(x) is an intensity at a pixel x, and ^(T) is atranspose operator.
 10. The method of claim 1, further comprising:repeating the selecting to determine all the points of the paraboliccurvature.
 11. The method of claim 10, further comprising: determining apose of the object based on the points of the parabolic curvature. 12.The method of claim 11, further comprising: acquiring templates ofparabolic curvatures determined by rotating the points of the paraboliccurvature; and determining an optimal transformation parameters aligningthe templates with points of the parabolic curvature.
 13. The method ofclaim 11, further comprising: determining similarities between thetemplates and the points of the parabolic curvatures using a chamfermatching technique.
 14. The method of claim 11, further comprising:refining an estimation of the pose using a Gauss-Newton optimization.15. The method of claim 10, further comprising: determining a type ofthe object based on the points of parabolic curvature.
 16. A system fordetermining points of parabolic curvature on a surface of an objectarranged in an environment, wherein the object is a specular object,comprising: a camera-object pair configured to acquire a set of imagesof the object under a relative motion between a camera-object pair andthe environment; and a processor configured to determine directions ofimage gradients at each pixel of each image in the set of images,wherein pixels from different images corresponding to an identical pointon the surface of the object forms corresponding pixels, wherein theprocessor selects the corresponding pixels having substantially constantthe directions of the image gradients as pixels representing a point ofthe parabolic curvature.
 17. The system of claim 16, wherein therelative motion between the camera-object pair and the environment isachieved by moving arbitrarily at least one of the camera-object pairand the environment.
 18. The system of claim 16, further comprising:determining a pose of the object based on the points of the paraboliccurvature.
 19. The system of claim 16, further comprising: determining atype of the object based on the points of parabolic curvature.
 20. Thesystem of claim 16, wherein the surface at the points of the paraboliccurvature curves only in one direction.